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Introduction 


The Division of Adult and Technical Education, Office of Career, Technical, and Adult 
Education (OCTAE), sponsors individualized technical assistance to support states in 
improving the quality of their Carl D. Perkins Career and Technical Education Act of 2006 
(Perkins IV) accountability systems. In response to OCT'AE’s October 2015 request for 
proposals, staff from the Nebraska Department of Education (NDE) shared that the state 
collects career and technical education (CTE) data far in excess of what is required for 
Perkins IV accountability purposes. These federal and state data are posted on the NDE 
website to publicize the contribution that CTE makes to students’ educational attainment 
and to support local program improvement efforts. State staff requested help in 
strengthening Nebraska’s current approach to disseminating CTE data and developing new, 


more effective strategies for disaggregating and analyzing data to augment state tools. 


RTI International researchers met with NDE staff beginning in November 2016 to discuss 
state needs and arrange for the transfer of program information. Project activities were 
temporarily suspended in late November to coincide with the scheduled leave of a key state 
staffer and resumed in February 2016 with a series of webinars that explored strategies for 
sharing information. The services of Sledge Creative, a website design and communications 
firm, also were engaged to create innovative, user-centric approaches for publicizing data. A 
memorandum of understanding between the state and RTI was executed in April 2016 to 
provide for the transfer of district-level Perkins IV accountability data, which were used to 


provide context for project work. 


This white paper summarizes the technical assistance services rendered to Nebraska and 
offers strategies the state may adopt to enhance users’ website experience and improve the 
dissemination of state and local CTE performance data. The paper opens with a description 
of website challenges associated with the state’s current approach to publicizing CTE data, 
followed by recommendations for improving the site’s design. Strategies for disageregating 
performance data to support local providers in identifying programs for improvement also 


are provided. 
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Existing Dissemination Approaches 


Nebraska is dedicated to sharing information on the type and characteristics of CTE 
programs offered throughout the state, as well as the demographics and performance 
outcomes of students enrolled. Information is disseminated using the state’s Data + 
Research + Analytics Website (DRAW), which is listed as a resource within the Nebraska 


Career Education home page. DRAW offers users quick access to empirical information on 


CTE and an extensive menu of options organized within three thematic areas. Information 
ranges from background materials describing the state’s approach to CTE accountability to 
statistics documenting state and local performance on the Perkins IV metrics. For example, 
within the Analytics section, users can click on the Statistics tab to obtain a detailed, 
statewide profile of students participating in CTE programming or they can click on the 
Perkins Indicators tab to view statewide average performance for the 2007-10 program 
years. District-level performance on the Perkins IV indicators is available for the 2009-10 
through 2014-15 program years, organized by sites’ Educational Service Unit (ESU) 
affiliation. These data include the percentage of CTE concentrators within districts that met 
the statewide performance goal for each metric, along with counts of the total number of 
CTE participants and concentrators within the ESU, disaggregated by race/ethnicity and 


cateer cluster. 


As an initial activity, RTI researchers examined the content and organization of DRAW. 


They identified the following three issues for state consideration: 


1. Complexity 


Nebraska collects a great deal of information on CTE programming and, to 
Ata communicate this wealth of data, has created a complex website that may present 
challenges to some visitors. Upon entering the Nebraska Career Education home 

page, visitors are presented with a sizeable menu of resources alphabetized by content. 
Visitors seeking performance data may initially click on the second option— 
Accountability—trather than the DRAW page, where accountability data are posted. 
Similarly, when entering DRAW, users are presented with a large menu of options, some of 
which are not clearly defined or intuitive. This can mean that new users must click through 


vatious links on the site in order to find the specific information they seek. 


To find statewide performance data on the Perkins IV indicators, users must click the 
Perkins Indicators link posted in the Analytics section. Information on district-level 
performance, in contrast, is accessed through the Report Cards link located in the site’s Data 


section. The site also presumes that users have some level of CTE technical knowledge; for 
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example, measures are frequently referenced using an accountability shorthand that may not 


be meaningful to those unfamiliar with the Perkins IV legislation (e.g., 1S1 for Academic 


Attainment in Reading/Language Arts). 


Graphical presentations also contain a 
substantial amount of information that, in some 
cases, is not presented accurately. For example, 
trend data on statewide student performance on 
the Perkins IV measutes, charted as a line graph, 
appear to fluctuate before falling precipitously 
on the nontraditional measures (Exhibit 1). This 
depiction is not technically correct, since each 
of the measures is discrete and, in some cases, 
includes different populations of students. 
Moteover, due to the way that the 
nontraditional measures are constructed (i.e., 
6S1 & 682), substantially lower levels of 
performance might be expected. Hlustrations 
included in the graphic (e., the blue circles) 


also are not explained. 


2. Aggregation 


Exhibit 1: Graphical Illustration of Data 


07-10 Secondary Longitudinal Average Performance 


a Toye) 


SOURCE: Nebraska Department of Education. 


State and district performance data within DRAW summarize the results of all 


students, by measure, or are disaggregated for subpopulations. While this is an 


effective approach for providing high-level, generalized information on statewide 


performances, aggregate data can mask underlying variation across districts, student 


populations, and/or CTE clusters. Consequently, statewide performance data are primarily 


useful for profiling the overall impact of CTE on students but are less actionable at the 


district level, where a finer grain of interpretation is needed. 


Users seeking district-level performance data may access the Report Cards section within 


DRAW. Here, information is arrayed at the ESU level, with data presented as the percentage 


of students achieving outcomes for each Perkins IV indicator. However, because districts are 


grouped regionally, it is difficult to compare performances across sites without accessing 


multiple files (and knowing in which ESU a comparison district is located). Moreover, 


because data are provided as percentages, the number of students included in the 


denominator of each measure is not available (Exhibit 2). This can complicate the 


interpretation of outcomes, since small population sizes can cause performance levels to 


fluctuate substantially with small changes in student performance. Educators also cannot 
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differentiate the performance of district Exhibit 2: District Performance Data, by ESU 


subpopulations, since information is only 


available in the aggregate. 13-14 PERKINS REPORT GARD 
Nebraska Department of asain. — 
Career Education - District Performance Report 

3 e B Uu rd e n DISTRICT NAME 1S1 182 281 3S1 481 $81 6S1 6S2 


Allen Consolidated Schools 70.00 70.00 | 0.004 | 100.00 | 100.00} 100.00 13.89 14.29 
Bloomfield Community Schools 100.00 | 100.00 | 100.00 | 100.00 | 100.00] 100.00} 40.00] 40.00 
Coleridge Community Schools 100.00 50.00 | 100.00 | 100.00] 100.00} 100.00} 39.29} 50.00 
Creighton Public Schools 95.00 | 100.00 | 100.00} 100.00] 100.00} 100.00] 53.76} 61.90 
Crofton Community Schools 89.66 | 100.00 | 100.00} 100.00] 100.00 96.15 55,65 37.93 
Emerson-Hubbard Public Schools 61.54 76.92 | 100.00 | 100.00 | 100.00} 100.00] 43.48 0.00 
Hartington Public Schools 50.00 42.86 | 100.00 | 100.00} 100.00] 100.00} 31.07] 60.00 
Homer Community Schools 83.33 66.67 | 100.00 | 100.00} 100.00] 100.00 7.69 0.00 


SOURCE: Nebraska Department of Education. 


Amassing the information included on 


DRAW tequires a substantial investment 


of time by state CTE analysts. Creating the 


many and varied exhibits requires that data be 


compiled and analyzed across years and then 


formatted for graphical display. Staff also must 

invest time to create cross-tabulations of student participation for each of the 16 cluster 
areas. Performing this work on an annual basis places a considerable burden on state staff 
members, which may be difficult to sustain given the many demands on their time. Indeed, 
the current statewide data posted on DRAW date to the 2009-10 program year, which limits 


their use for program improvement purposes. 


In sum, while Nebraska has made a commitment to publicizing CTE information to the 
public and field, DRAW’s organizational complexity, high level of aggregation of district and 
student data, and staff time required to curate content undermine the website’s utility. There 
are, however, clear benefits to sharing information with educators, and the state has laid a 
solid foundation upon which to build. With some modest refinements in how DRAW 
content is organized, accessed, and displayed, the site can be transformed to provide users 


with timely, actionable data that is tailored to fit their individual needs. 


Data Dissemination Strategies 


Reorganizing Nebraska’s DRAW begins with recognizing the multitude of visitors 

who will access the site. Potential users may include state education 

administrators, district administrators and CTE coordinators, teachers and 
counselors, students and parents, and the public and press. Meeting the needs of this diverse 
group will entail configuring the website to be more intuitive for first-time users and those 
lacking extensive background in CTE accountability. At the same time, individuals with 
more advanced knowledge must be able to quickly navigate the site to find the information 


they need. 


DATA DISSEMINATION STRATEGIES FOR CAREER AND TECHNICAL EDUCATION IN NEBRASKA 
TECHNICAL ASSISTANCE TO STATES PROJECT 5 


Consumer-Centered Design 


> During technical assistance conversations, state staff members shared that their 


als initial focus was on providing Nebraska state and local administrators and staff 
with access to statewide and district-level CTE performance data. While the 

majority of DRAW visitors may be those with some CTE background, educators lacking 
such knowledge also will make use of the site. For example, efforts to expand the career 
readiness of all high school students, and not just those in CTE programs, may attract an 
increasing number of academic educators. District staff new to CTE also will need some 
level of introduction. Finally, with the upcoming reauthorization of Perkins IV looming, it is 
likely that all educators—CTE and otherwise—will require a refresher to keep pace with 
anticipated changes to federal CTE accountability requirements. Given the potentially large 


site demands, state staff may not wish to limit their audience in undertaking a site redesign. 


Exhibit 3: Website Design Plan 


NE CTE timeline wireframe 
+ for work function + flow 
DRAW + milestones layout 
existing + intended roles + collect resource 
user groups responsibilities + write curation 
> relative priority information + updates 


SOURCE: RTI International. 


As a first step, the state should create a website design plan that identifies existing and 
intended user groups and their relative priority; lays out website goals and a strategic 
direction, for both Nebraska Career Education and DRAW; delineates the roles and 
responsibilities of staff participating in the redesign effort; and outlines a timeline for 
accomplishing work. This action plan should include key milestones, including the collection 
of information to help decide site components, initial design and layout recommendations, 
the creation of site witeframes, and resource needs to maintain the site once it is launched 


(Exhibit 3). 


Ensuring that the site meets user needs (internal and external) and disseminates the 
appropriate information is a critical step in the redesign process. This will entail soliciting 
input from potential users to determine the content that should be included on the website 
and the manner in which it should be incorporated. The state has several options for 


gathering feedback—none of which are mutually exclusive. These include the following: 
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© = Create a project workspace within NDE—tely on state staff members to identify local 
needs informally, drawing on their expertise and knowledge of field information 


requests. 


e = Perform a competitive analysis—state staff members could look at other state sites and 
compare strengths, weaknesses, and highlight areas they may wish to draw upon (as 


well as areas/designs they would want to avoid). 


© = =Review website analytics—state staff could do a website audit of key metrics (e.g., 
number of visitors, most visited pages, which data are being viewed the most, etc.) 


to help determine the inclusion/exclusion of any elements in a redesign effort. 


© = Interview potential user groups—tepresentatives from key stakeholder groups could be 
contacted to identify their data needs and desired style of communication of 


information. 


e Host listening sessions—state staff could hold regional meetings or attend the annual 
conference of Association for Career and Technical Education of Nebraska to 


solicit input from CTE administrators and educators. 


© Convene a task force—state CTE representatives, selected from throughout the state, 
could be convened, using an internal or external facilitator, with website design 


expertise, to guide discussions. 


Once information has been compiled, the state should hold a synthesis session to review 


identified and connected insights and array action items to guide project work. 


Simple, Functional Design 


Care should be taken that DRAW does not overwhelm users with information. 

Feedback from potential users can help staff prioritize the data and resources that 

are presented, as well as how they should be organized. In lieu of the current 
organization of data, research, and analytics, Nebraska may wish to consider structuring 
DRAW to present information in a hierarchical format, beginning with broad, statewide 
insights that may be meaningful for all users. Those seeking more detailed data would then 


access tools that would allow them to construct their own analyses. 


Creating this layered approach may entail consolidating accountability resources now 
appearing on the Nebraska Career Education and DRAW Web pages, as well as 
restructuring those contained within DRAW. For example, DRAW could begin with a high- 
level overview of the performance expectations attached to CTE programming, followed by 
the state indicators and definitions used to measure district and student performance. The 


“exploded view” of the state’s Secondary Perkins Data Manual, currently included in the 
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Indicators tab on the Nebraska Career Education Web page would be an excellent resource 
for this purpose. State and local performance data could then follow. Users new to CTE 
would thus have a context for understanding the performance data that followed, while 


those with advanced knowledge could skip to more detailed information. 


Irrespective of the site’s final organization, the state may wish to hire an experienced website 
design firm, ideally one with background in CTE issues, to support state staff in redesigning 
DRAW. This can help ensure that the site is constructed using design principles that will 

maximize usets’ experience, in part by providing them with an intuitive understanding of site 


navigation and organization. 


Moving from Data to Insight 


re Y_ Akey benefit of DRAW is its capacity to highlight CTE 

2 QO: performance data that warrant special attention. For example, NDE 

staff have used the Perkins Indicators within the Analytics section 

of the website to illustrate the performance of subpopulations of students participating in 
CTE instruction. These data can be used to craft a compelling storyline that helps users 
understand the importance of CTE and the contribution it makes to student outcomes. 
These pre-populated graphics can serve as an executive summary on CTE accountability. 
Key statewide takeaways can be presented to the user to illustrate the contribution that CTE 
makes to students’ success in a way that aligns with the state’s overall goals for program 
delivery. Ideally, users will take the information and structure their own analysis of their local 


data. 


Packing a significant amount of information into a single figure can overwhelm users, 
particularly those who lack an advanced understanding of the Perkins IV metrics and/or a 
statistical background. One solution is to shift the website focus from simply posting 
statistical data to sharing insights. This can be accomplished by adopting strategies to engage 
users though short descriptions and images that relate and contextualize the data. One 
means of doing so is to add a human face to the outcomes, for example juxtaposing statistics 
with photos of students (Exhibit 4). 
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Exhibit 4: From Data to Insights 


Females were five timesmore likely'than.males to. complete’a CTE 
relceye [e-lanuiam-mileemareyaler-telidte)ar-lmiolmuaal-l] me) -lele(-1¢ 


N 


07-08 Performance Nontraditional Completion (State Level) 


Females were five times more likely 
idaYclamaatsl(=smKenee)gn] ©) (18 


0.00% 10.00% 20.00% 30.00% 40.00% 


Special education youth recorded the 
lowest completion rates. 


Native Americans posted the highest 
rate of completion. 


SOURCE: RTI International. 


Data headlines also can be added to draw readers’ attention to the content and provide a 
context for understanding information. Such graphical storytelling can help users interpret 


statistics, as well as find their own story in the data. 


Customization: Empower Users to Design Their Report 


ales Users will visit DRAW with different needs. Some will be seeking to learn more 
e— = about the statewide contribution that CTE makes to students’ success, while 
others will want more targeted information, such as how students in a particular 
district, school, or program are performing. Accordingly, data should be provided in a way 
that is accessible by a wide audience. Variables for analysis should be prioritized and clearly 
delineated, analytic choices uncomplicated and free of statistical jargon, and findings offered 
in varying formats to accommodate individuals with different preferences for viewing data. 
Moreover, since users will have differing content background, ranging from those with little 


knowledge of CTE (e.g., the press, parents) to those with advanced understanding (e.g., CTE 
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teachers and school administrators), links to terminology should be provided so that users 


correctly interpret what the data represent. 


Ultimately, the design of DRAW should reflect the input received from its potential users. 
While it is premature to provide specific recommendations on how the analytic component 
of the site should be constructed, Nebraska may wish to create a top-level “pick menu” that 
allows users to select the data they wish to explore, at their desired level of specificity.! This 
approach offers several benefits. One is that it reduces site complexity because all users 
funnel through a single point of entry that can be prominently featured. Site maintenance is 
also reduced, since data requests will hub-and-spoke off a single page. Another benefit is that 
users may discover new questions or think of better ways of accessing the data when viewing 


the analysis options available to them. 


Ideally, the variables and statistical tools included in the pick menu will enable the majority 
of users to answer any questions they may have about CTE program performance and 
student outcomes—be they at the state or district level. While it is possible to integrate 
sophisticated analytic tools into the pick menu, for example to allow users to perform 
multiple regressions or generate matched comparison groups, providing such options can be 
expensive and may lead individuals to misinterpret results.* For this reason, RTI 
recommends that the state keep its web analytic tools simple, intuitive, and comprehensible. 
Users seeking to perform more complicated analyses should be directed to contact state staff 


for access to data, which typically will be run at the individual student level. 


In RTPs experience, the needs of most users will be satisfied with descriptive statistics, such 
as the percentage of CTE concentrators achieving state academic standards or proportion of 
students enrolling in different program areas. Cross-tabulations will allow users to 
disageregate data to address targeted issues, for example, to identify the percentage of male 


and female CTE students who graduated from high school within four years of their entry. 


Identification of variables for pick menu inclusion should reflect the information gathered as 
part of the state strategy employed to identify user needs. Some menu options are likely 
predetermined given federal reporting requirements, including such variables as student 
demographics, the characteristics of education districts, and federal Perkins IV reporting 


requirements. Others will be at state discretion. Ideally, pick menus will be designed to allow 


! For an example of how such a pick menu might operate, see the Perkins Data Explorer housed on the Perkins 


Collaborative Resource Network. This innovative tool enables users to create customized Perkins IV enrollment 
and performance reports using statewide data. 

? For example, RTI designed DATALAB for the National Center for Education Statistics to enable data users to 
generate complex analyses, such as correlation matrices and logistic regression of variables housed in federal 
datasets. See http://nces.ed.gov/datalab/#2. While intended for public use, users must have considerable 
statistical knowledge to correctly specify and run analyses. Development costs also may make such an approach 


infeasible at a state level. 
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individuals to “see themselves” in the state data, meaning that they can find variables that 


mirror their own status and programs, as well as identify sites to which they may compare 


themselves. While the variables contained in Exhibit 5 primarily are intended for illustrative 


purposes, the listing of districts is critical if individuals are to assess their performance 


relative to other sites or to disageregate information for comparative purposes. 


Exhibit 5: Pick Menu 


My Account | Log Out 
Home Management Reports Custom Reports Collection Dashboarc File Cabinet Administrative Tools Links 

District: Year: Student Populations: 

All all = © al - 

Adams Central Public Schools 2015 Gender a 

Ainsworth Community Schools 2014 = ] Male 

All Saints Catholic School F] 2013 Female 

len Consolida ool . 
Allen Consolidated Schools ~J 2012 Race/Ethnicity (1997) 
Alliance Public Schools 2011 


Alma Public Schools + 


Core Indicators: 


Co all 


m 


SECONDARY 


( 1S1: Academic Attainment in 
Reading/Language Arts 


©) 182: Academic Attainment in 
Mathematics a 


2010 ~ 


Student Categories: 


All 
Special Populations 


Disability Status 
(ESEA/IDEA) 


Displaced Homemakers 


Economically 
Disadvantaged 


} American Indian or Alaskan 


Data Options: 


All 


m| > 


Numerator 


Actual 


Denominator 


Target/FAUPL 


Performance 


Met 90% 


Contact: CTEhelp@nebraska.gov 


SOURCE: RTI International. 


Other variables for pick menu inclusion could include the following: 


e District Characteristics—information on the geographic location or features of 


the students or community in which it is housed. This could include information on 


site urbanicity (Le., urban, suburban, rural), size of student enrollment, or the 


socioeconomic status of the population served. 


e Program Offerings—types of CTE programs provided within the district, with 


options ranging from high-level categories, such as career field, to more targeted 


fields, including career cluster or a specific CTE program. 


In populating pick menus, state staff should choose criteria based on identified state policy 


goals, as well as the validity and reliability of the data used to make assignments. Care also 


should be taken to not overwhelm individuals with information. For example, while it may 


be possible to include all of the student subpopulations referenced in Perkins IV, 


identifications such as single parents (including single pregnant women), may not offer 


accurate data. The state also may wish to exclude variables for which there are small 
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populations of students, since results may be subject to large percentage swings due to small 


performance changes. 


Actionable: Analysis Must Produce 
Meaningful Data 


Website visitors will use CTE performance data for differing purposes. Analysis of 
@) statewide data will produce results that are highly agereeated—cutting across all 

CTE students in all districts and programs. This type of data can be used to 
message the overall contribution that CTE makes to students’ educational progress and the 
state economy. For example, data on CTE high school graduates’ enrollment in 
postsecondary education or placement into employment can be used to highlight the role 
that technical education makes in preparing students for life transitions. When packaged as 
insights, these outcomes can help to communicate the benefits that CTE confers and, in so 


doing, better inform the public’s perception of CTE programming. 


Users must have a context for interpreting CTE performance outcomes if statewide data are 
to prove useful at the local level. Simply put, educators must be able to view their own 
program and student performance results and compare them against those of others in the 
state if they are to objectively gauge their instructional effectiveness. The state has several 
options for helping districts to benchmark their data, all of which could be easily 


incorporated into the pick menu described above. 


Finding Districts Like Me 


While Nebraska’s ESU organization offers both an administrative and 
e geographical context for interpreting CTE data, the districts that comprise an 
ESU vary along a number of dimensions. For example, the number and 
demographics of students participating in CTE may vary across sites, as may the type and 
scale of programming offered. Consequently, districts with similar characteristics may be 
scattered across the state. To assist educators in finding comparison sites, the state could 


provide districts with options for finding “districts like me.” 


To illustrate, staff from a rural school district might choose to compare their performance 
on technical skill assessments against that of other rural districts, identified using a 


classification system designed by the state. Sites could then have the option of seeing their 
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own performance data arrayed against similar sites, as well as in relation to the state as a 
whole (Exhibit 6). 


Exhibit 6: Districts Like Me: Percentage of Students Passing Technical Skill Assessments (2S1), 
by District Type: 2014-15 


100% 


g2% 89% 82% 


80% 


60% 


40% 


20% 


0% 
District Comparison Rural Districts State 


mEconomic Disadvantaged mDisabled mEnglish Language Learner 


SOURCE: RTI International. 


Users could employ the pick menu options to further unpack their district performances to 
surface additional information that might otherwise be masked. For example, while the 
featured district in Exhibit 6 appears to be relatively more successful in supporting English 
Language Learners (85 percent vs. 68 percent in other districts), this rate may be less 
meaningful if the district were serving a relatively small number of students. For this reason, 
the state may wish to consider providing users with options for selecting one or more 
districts against which they may compare their performance. To aid in interpretation, the 


number of students included in the denominator of each measure could also be provided. 


There are, however, limits to the amount of disaggregation that is feasible. Given small 
district sizes throughout the state, it may be counterproductive to control for more than one 
population characteristic at a time. For example, while it might be hypothetically possible to 
cross-tabulate gender with special population status, in most districts this would produce 
only a handful of students for most cells (e.g., females who are economically disadvantaged). 
These data would be unreportable due to cell suppression designed to prevent personally 


identifiable information from being released. 
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Performance by School 


r While many districts within Nebraska comprise a single high school, larger 
Fro cities may have several high schools operating within a district. For example, 
Exhibit 7 details a fictitious district that offers CTE programming in each of two high 
schools. While district performance on the 6S2 measure suggests that roughly 60 percent of 
nontraditional students are completing their CTE programs—a relatively high proportion 
compared to the statewide average of 30 percent—examination of performance rates by 
gender indicates that only 44 percent of males are completing nontraditional programs, as 


compared to 75 percent of females (Exhibit 7). 


Exhibit 7: Nontraditional Completion Rates by Gender for Unnamed District 
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SOURCE: RTI International. 


While district-level data can assist district administrators in assessing outcomes across 
student groups, additional information may be afforded by further disageregating district 
data by the high schools that comprise it. Exhibit 7 offers an example of how school-level 
data could provide additional information to support program improvement efforts. 
Specifically, data indicate that district performances are being negatively affected by 
performance outcomes in High School 1. Differential rates of performance may be due to a 
range of issues, from the quality of instruction to the mix of CTE programs offered within 
schools. It may be, for example, that one school offers a predominance of programs that are 


strongly gender specific, which may make it more difficult to raise student performance. 
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Performance by Program 


Nebraska districts offer a range of CTE programming that crosses the six career 
C) fields and 16 clusters recognized by the state. However, since federal CTE 
“—* reporting requirements are focused at the statewide and individual district level, 
performance variations across fields and clusters (and programs within clusters) may be 
masked. This may sacrifice potentially useful information, since performance results for 


large, high-achieving programs may eclipse those of smaller, lower-quality programs. 


Exhibit 8 documents how district data might be unpacked to assess the relative performance 
of students enrolling in differing CTE programs. On average, roughly half of students 

(52 percent) met the state standards for attainment of academic skills in reading/language 
arts (1S1). However, student performance on the measure varied by cluster, with students in 
the science, technology, engineering, and mathematics (STEM) program nearly three times 
as likely to achieve state standards relating to reading and language arts as those in the 
manufacturing cluster (80 percent vs. 28 percent, respectively). Overall, seven programs 


recorded performance rates below the district average (Exhibit 8). 


Exhibit 8: District Performance Across CTE Programs for Measure 181—Academic Attainment 


Numerator Denominator Pertormance 


MEASURE 1S1;ACADEMIC ATTAINMENT 


Agriculture, Food, and 14 45 31% 
Natural Resources 
Systems 
* Architecture and 86 126 68% 
Construction 
Arts, Audio/Visual, and 1 21 52% 
Communication 
Business Management and 121 244 50% 
’ Administration 
Education and Training 34 51 67% 
™ Finance 30 A 73% 
Health Sciences 217 431 50% 
Hospitality and Tourism 149 335 44% 
Human Services 124 266 47% 
5. Information Technology 180 345 52% 
Law, Public Safety, Bil 92 55% 
= Security, and Corrections 
, Manufacturing 37 130 28% 
a Marketing 101 145 70% 
Science, Technology, 97 122 80% 
Tare llatexeualaremelare| 
Mathematics (STEM) 
Transportation, Distribution, 14 48 29% 
AY VEeTCclaTolersii are Mare lave Mele] isy(Cers} 


SOURCE: RTI International. 


A range of factors may explain these differing programmatic results. For example, the 
relatively higher performance of students in the STEM cluster could be due to the type of 
students who choose to concentrate in the field, rather than the quality of the instruction 
offered. Irrespective of the cause, educators focused on district-level results may overlook 
issues within clusters (or programs within clusters) that may warrant attention. Again, further 
disaggregation of cluster-level data, such as controlling by student race/ethnicity, gender, or 


special populations status may surface additional information. 
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In addition to comparing intra-district performances across clusters, educators may benefit 
from comparing their CTE field, cluster, or program-level data to that of other districts 
offering similar programming. For example, Exhibit 9 offers fictionalized performance data 
for the nontraditional completion measure, controlling for students participating in the 
manufacturing cluster within the Skilled and Technical Sciences field. In practice, such a 
table could be created for all career clusters and include information for all districts offering 


the program. 


Exhibit 9: Performance of Students Preparing for Nontraditional Occupations 


MEASURE 482: FEMAEE*PARTCIPATION IN 
NONTRADITIONAL OCCUPATION FIELDS 


District performance on the nontraditional completion measure (4S2) for females 
participating in career fields and clusters that prepare them for occupations 
nontraditional for their gender: 2013-14 
District Name Total Non- District Statewide Percentage 
traditional Perform average Point 
student ance performance Difference 
enrollment level 


Skilled and Technical Sciences: Manufacturing 


01-00xx District A Public 51 
Schools 
78-00xx _ District B Public 
Schools 
82-00xx District C Public 
Schools 
89-00xx _ District D Public 
Schools 
89-00xx _ District E Public 
Schools 


SOURCE: RTI International. 
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While a variety of information could be included as columns in the table, analysis is intended 
to support district staff in pinpointing other sites offering similar programming, the relative 
number of students participating in the program, and the district performance level. This 
information can assist educators in disentangling the effect of other programs on CTE 
performances and, in so doing, offer a more refined understanding of the relative impact of 
their programming. In the case of this measure, a comparable set of tables could be 
produced for males participating in clusters that prepare them for career entry into 


occupations nontraditional for their gender. 
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Conclusion 


© The state of Nebraska collects a substantial amount of information on CTE 
students’ performance on the federal Perkins IV indicators, as well as additional 
state metrics. The state currently disseminates this information using DRAW, which is 
housed as a resource on the Nebraska Career Education website. Unfortunately, the utility of 
state CTE data are compromised by the manner in which data are organized, analyzed, and 
maintained. The state has an opportunity to redesign its website to simplify the way that 
information is shared with the field and how data can be accessed to have greater meaning at 


the local level. 


Website development efforts should be geared towards creating a simple, functional, user- 
defined experience. Site materials should be organized in a straightforward, intuitive manner 
so that visitors can easily navigate the site to find the information that they seek. Design 
efforts should begin with the creation of a detailed action plan, laying out the goals of the 
site, roles and responsibilities of key staff, and a timeline and milestones for site completion. 
Representatives of the field also should be engaged throughout the process to ensure that 
the site is designed to meet their needs. The state may wish to consider hiring a web design 


firm to ensure that the site includes best practices in displaying data. 


Rather than seeking to pre-populate all the content that users might seek, state staff may 
wish to create a pick menu or other tools to allow users to conduct their own exploration of 
the data. Moreover, while aggregate data are useful for communicating the benefits that CTE 
can confer, data must be actionable at the local level if they are to inform program 
improvement efforts. For this reason, the state should provide users with access to district- 
level data and tools for comparing outcomes across sites, so that the relative success of 
programs can be determined. Users also should have the option of disageregating data by 
key student populations, in particular those defined in the Perkins IV legislation, so that 


educators can assess the relative performance of different student populations. 


